Skip to content

feat(vast): add Vast.ai SSH lease provider#680

Merged
steipete merged 31 commits into
openclaw:mainfrom
coygeek:vast-provider
Jul 3, 2026
Merged

feat(vast): add Vast.ai SSH lease provider#680
steipete merged 31 commits into
openclaw:mainfrom
coygeek:vast-provider

Conversation

@coygeek

@coygeek coygeek commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes #363

  • Add the Vast.ai direct Linux GPU SSH-lease provider with config/env/flag handling, provider registration, ownership labels, per-lease SSH keys, guarded offer selection, and create/use/release/cleanup lifecycle support.
  • Bind destructive cleanup to the exact local claim, Vast account, instance, and credential-free API endpoint; fail closed on missing or changed ownership, inventory errors, ambiguous SSH keys, and endpoint/account mismatches.
  • Reconcile Vast's documented and live API response shapes, including v1 pagination, JSON-encoded SSH-key lists, success-only mutation responses, and post-delete 200 {"instances":[]} responses.
  • Bootstrap required SSH tooling, preserve exact cleanup routing through runtime state updates and generated stop commands, and add billable live-smoke coverage with zero-residue assertions.
  • Add VISION.md lifecycle boundaries and comprehensive Vast provider documentation. Changelog credit retained for @coygeek.

Verification

Exact head: bd051ada04060feb14bc482b7dd7d8ce2fc11a13.

  • go vet ./...
  • go test -race ./...
  • node --test scripts/live-vast-smoke.test.js — 10 passed
  • bash -n scripts/live-vast-smoke.sh
  • scripts/check-docs.sh — 51 command docs, 70 providers, 203 Markdown files, generated docs site
  • git diff --check
  • Structured whole-patch autoreview is clean with no accepted/actionable findings.
  • Exact-head public CI is terminal green: https://github.com/openclaw/crabbox/actions/runs/28648802379

Authenticated Vast proof

Run against the exact head above with the guarded $0.50/hour cap, one required GPU, destroy release policy, and nvidia/cuda:12.4.1-base-ubuntu22.04 to avoid the development image's long pull:

  • authenticated doctor: auth=ready control_plane=ready inventory=ready leases=0
  • created owned instance 43686387, lease cbx_9ad23b697061, RTX 5070 Ti
  • SSH bootstrap completed and nvidia-smi -L proved one GPU
  • list/status retained the exact credential-free endpoint https://console.vast.ai/api/v0
  • detached the owned SSH key and destroyed the instance
  • final proof: pre_owned=0 post_owned=0 cleanup=complete; authenticated doctor again reported leases=0
  • local claim and per-lease key material were removed

Failure-path canaries also proved rollback after rejected create payloads, live SSH-key response drift, a success-only mutation response, and an intentionally aborted slow image pull; every attempt returned to zero owned instances with no retained local claim or key material.

@clawsweeper

clawsweeper Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Codex review: found issues before merge. Reviewed July 3, 2026, 4:48 AM ET / 08:48 UTC.

Summary
The branch adds a built-in Vast.ai Linux SSH-lease provider with config/env/flags, provider registration, REST lifecycle, guarded cleanup, docs, metadata, tests, and live-smoke coverage.

Reproducibility: not applicable. This PR adds a new provider rather than fixing a current-main bug. The relevant verification path is the live Vast smoke, and the PR body reports an authenticated exact-head create/use/destroy run.

Review metrics: 2 noteworthy metrics.

  • Diff surface: 36 files changed, +5,226/-13. The branch spans config, provider lifecycle, docs, scripts, tests, generated provider metadata, and one Worker test config path.
  • Live Vast proof: 1 authenticated create/use/destroy run reported. The external billable provider boundary depends on real offer creation, SSH readiness, GPU execution, cleanup, and final empty inventory proof.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #363
Summary: This PR is the implementation candidate for the open Vast.ai direct GPU SSH-lease provider request.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🐚 platinum hermit
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Get explicit maintainer acceptance for the built-in billable Vast provider direction and cleanup contract.
  • Resolve whether the release-owned changelog entry should stay or move to PR-body release-note context.

Risk before merge

  • [P1] Merging adds a new credentialed, billable, destructive-cleanup provider boundary; maintainers still need to explicitly accept Vast as a built-in first-party provider.
  • [P1] The live proof in the PR body is sufficient copied CLI output, but maintainers may still prefer a fuller redacted transcript for the first merge of this provider.
  • [P1] The PR edits release-owned CHANGELOG.md, so release-note handling needs a maintainer decision before merge.

Maintainer options:

  1. Confirm first-party Vast acceptance (recommended)
    A maintainer should explicitly accept the built-in billable Vast provider, defaults, credential boundary, and destructive cleanup contract before merge.
  2. Accept the live-smoke evidence
    Maintainers can treat the copied exact-head authenticated smoke output as enough proof and own any remaining Vast API drift risk.
  3. Pause the provider direction
    If Vast should wait for a broader provider policy decision, pause or close this PR and keep Add Vast.ai as a direct GPU SSH-lease provider #363 as the canonical tracker.

Next step before merge

  • [P2] Remaining action is maintainer product and safety review for a new billable first-party provider, plus changelog ownership handling; this is not a safe automated repair lane.

Security
Cleared: No concrete static credential leak, auth bypass, or cleanup-ownership bypass was found in the diff; the new security-sensitive provider boundary still needs maintainer acceptance.

Review findings

  • [P3] Remove the release-owned changelog edit — CHANGELOG.md:7
Review details

Best possible solution:

Land after maintainers accept the built-in Vast provider scope and cleanup contract, and resolve whether the release-owned changelog entry stays or moves to PR-body release-note context.

Do we have a high-confidence way to reproduce the issue?

Not applicable; this PR adds a new provider rather than fixing a current-main bug. The relevant verification path is the live Vast smoke, and the PR body reports an authenticated exact-head create/use/destroy run.

Is this the best way to solve the issue?

Unclear: the adapter shape matches existing direct SSH-lease provider patterns and the live proof now covers the key lifecycle, but first-party billable-provider acceptance and release-owned changelog handling need maintainer judgment.

Full review comments:

  • [P3] Remove the release-owned changelog edit — CHANGELOG.md:7
    This PR edits release-owned CHANGELOG.md; normal feature PRs should keep release-note context in the PR body or commit messages unless a release owner intentionally keeps this entry.
    Confidence: 0.84

Overall correctness: patch is correct
Overall confidence: 0.82

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 79c985cbf114.

Label changes

Label changes:

  • add proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes copied exact-head live output for an authenticated Vast doctor/list/create/SSH GPU command/status/destroy/zero-residue run plus failure-path canaries.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body includes copied exact-head live output for an authenticated Vast doctor/list/create/SSH GPU command/status/destroy/zero-residue run plus failure-path canaries.
  • remove rating: 🧂 unranked krab: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.
  • remove status: 📣 needs proof: Current PR status label is status: 👀 ready for maintainer look.

Label justifications:

  • P2: This is a normal-priority first-party provider expansion with bounded blast radius but meaningful merge-safety review needs.
  • merge-risk: 🚨 security-boundary: The PR adds credentialed Vast API calls, endpoint validation, SSH-key handling, account binding, and destructive cleanup paths.
  • merge-risk: 🚨 other: Green static checks do not by themselves settle billable provider safety, live API drift, capacity behavior, or first-party provider acceptance.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body includes copied exact-head live output for an authenticated Vast doctor/list/create/SSH GPU command/status/destroy/zero-residue run plus failure-path canaries.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes copied exact-head live output for an authenticated Vast doctor/list/create/SSH GPU command/status/destroy/zero-residue run plus failure-path canaries.
Evidence reviewed

What I checked:

  • AGENTS.md policy read: The repository policy was read fully and applied to the provider-neutral adapter boundary plus no-secrets-on-argv review. (AGENTS.md:13, 79c985cbf114)
  • Current main lacks Vast support: A current-main search for Vast provider names returned no matches, so the PR is not obsolete on main. (79c985cbf114)
  • Latest release lacks Vast support: The same provider-name search against v0.34.0 returned no matches, so the provider is not shipped in the latest release. (ef898c27570a)
  • Provider registration matches the requested shape: The PR registers canonical provider vast as a Linux SSH-lease provider with SSH, crabbox-sync, cleanup, and CoordinatorNever. (internal/providers/vast/provider.go:23, bd051ada0406)
  • Credential and cleanup boundary inspected: The implementation validates cleanup against the stored Vast API endpoint and account identity before destructive release paths. (internal/providers/vast/backend.go:882, bd051ada0406)
  • Live behavior proof is now present: The PR body reports an exact-head authenticated Vast run covering doctor, empty inventory, create, SSH bootstrap, nvidia-smi, list/status endpoint retention, SSH-key detach, destroy, final empty inventory, and rollback canaries. (bd051ada0406)

Likely related people:

  • steipete: PR history includes later Vast fixes for account-bound cleanup, SSH-key reconciliation, cleanup identity preservation, and success-only response scoping; current-main provider interface blame also points to this area. (role: recent Vast safety contributor and provider architecture contributor; confidence: high; commits: bd051ada0406, b38cc1110f5a, 5ade62453c33; files: internal/providers/vast, internal/cli/provider_backend.go, scripts/live-vast-smoke.sh)
  • coygeek: Git history shows merged direct-provider work for DigitalOcean, NVIDIA Brev, and Hostinger, and this PR's initial Vast config/lifecycle commits follow those patterns. (role: feature proposer and prior direct-provider contributor; confidence: high; commits: aff04c9f19a3, 975d70260d7b, baa8562f139b; files: internal/providers/digitalocean, internal/providers/nvidiabrev, internal/providers/hostinger)
  • Yossi Eliaz: Git history shows the original RunPod provider commit, which is the closest existing GPU SSH-lease provider pattern. (role: introduced adjacent GPU SSH provider; confidence: medium; commits: cef985b92482; files: internal/providers/runpod)
  • Vincent Koc: Recent RunPod cleanup and review-finding fixes are relevant to lifecycle safety for another direct GPU SSH-lease provider. (role: recent adjacent lifecycle contributor; confidence: medium; commits: 35b728746e4e, ad32181f38b0; files: internal/providers/runpod)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal priority bug or improvement with limited blast radius. merge-risk: 🚨 other 🚨 Merging this PR has meaningful risk outside the owned taxonomy. labels Jun 25, 2026
@coygeek

coygeek commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper clawsweeper Bot added the merge-risk: 🚨 security-boundary 🚨 Merging this PR could weaken sandboxing, authorization, credentials, or sensitive data. label Jun 25, 2026
@coygeek

coygeek commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@steipete
steipete force-pushed the vast-provider branch 2 times, most recently from 8f13285 to 19523ad Compare July 2, 2026 19:12
coygeek added 18 commits July 2, 2026 23:13
Add the first-party Vast provider configuration surface with env-only API-key sourcing, safe config output, provider flags, built-in registration, and offline placeholder backend contracts for later lifecycle plans.

Add focused tests for Vast config precedence, credential destination provenance, provider discovery, flag registration, validation, and secret redaction.
Add the provider-local Vast.ai HTTP client, request and response models, payload builders, redacted API error handling, and compact ownership label helpers for the upcoming lifecycle implementation.

Cover the client contract with fake HTTP tests for auth, redirects, offer search, instance creation and management, SSH-key methods, response decoding, redaction, and label ownership safety.
Add the Vast SSH-lease backend for acquire, resolve, list, release, touch, doctor, and cleanup using generated per-lease keys and strict cbx1 ownership labels.

Cleanup is guarded by local claims plus provider labels, release destroys by default, and lifecycle tests use a fake Vast API/readiness path without live credentials.
Use fileURLToPath for the Cloudflare Workers mock alias so Vitest resolves the local mock correctly from repository paths that contain spaces.
Respect the generic SSH user override when Vast provider defaults are applied, so provider-specific defaults cannot replace an operator-supplied SSH user. Add regression coverage for both the Vast backend config and embedded direct SSH backend config.
Keep cleanup enabled for possible partial Vast warmup failures, but do not let a pre-acquire not-found stop response convert a known capacity blocker into cleanup_failed. Add regression coverage for the no-lease cleanup path.
Keep config show from replacing an explicitly supplied generic SSH user with the Vast provider default. This keeps the displayed effective config aligned with the backend defaulting path.
Report the release action captured on the lease target instead of the current config, so stopped or kept Vast instances are not reported as destroyed. Also route cleanup of owned-looking instances without matching local claims through the safe missing-claim refusal path.
Give local fake-SSH probe tests the same headroom as other terminal-bound checks so transient scheduling delays do not fail unrelated provider verification.
Preserve persisted Vast release and key metadata when normal resolve refreshes local claim endpoints, so stop/keep leases do not drift back to destroy. Also allow status-only resolution for owned instances that do not yet expose an SSH endpoint.
Detach provider-owned Vast SSH keys before destroying the instance so normal release cleanup follows the same effective ordering as rollback. Add an order assertion to prevent the detach call from becoming a suppressed post-destroy no-op.
Let an explicitly supplied Vast release action override the persisted claim label during release and reporting. This keeps safety overrides such as --vast-release-action keep from being ignored on leases acquired with the default destroy policy.
Give controller subprocess tests more non-race scheduling headroom after repeated local failures in terminal-bound process lifecycle checks. This keeps provider verification from failing on unrelated launch-gate timing.
Encode Vast search and create requests using the documented API shapes, restore stored SSH keys on resolve, and keep stopped lease claims so retained instances can be reconciled or destroyed later.

Also widen a race-instrumented Apple VZ helper test timeout encountered during the final verification gate.
Drop unused Vast helper functions left behind after the pagination and release-lifecycle fixes. CI runs deadcode with test reachability, and these helpers were no longer referenced by the provider implementation or tests.
Reject query strings and fragments in the Vast API URL so configured provider endpoints cannot smuggle secret-bearing URL components or break path construction. Document the endpoint contract and cover query and fragment inputs in provider validation tests.
@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 3, 2026
@steipete
steipete merged commit 878b370 into openclaw:main Jul 3, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 other 🚨 Merging this PR has meaningful risk outside the owned taxonomy. merge-risk: 🚨 security-boundary 🚨 Merging this PR could weaken sandboxing, authorization, credentials, or sensitive data. P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Vast.ai as a direct GPU SSH-lease provider

2 participants